Most video applications seek highest possible perceptual quality given the bit rate constraints. For instance, in a low bit rate application such as a videophone system, a video encoder may provide higher quality by eliminating the strong visual artifacts at the regions of interest that are visually more important. On the other hand, in a high bit rate application, visually lossless quality is expected everywhere in the pictures and a video encoder should also achieve transparent quality. One challenge in obtaining transparent visual quality in high bit rate applications is to preserve details, especially at smooth regions where loss of details are more visible than that at the non-smooth regions because of the texture masking property of the human visual system.
Increasing the bit rate is one of the most straightforward approaches to improve the quality. When the bit rate is given, an encoder manipulates its bit allocation module to spend the available bits to obtain the highest possible quality. In non-real-time applications such as DVD authoring, the video encoder can facilitate a variable-bit-rate (VBR) design to produce a video with a constant quality on both difficult and easy contents over time. In such applications, the available bits are appropriately distributed over the different video segments to obtain constant quality. In contrast, a constant-bit-rate (CBR) system assigns the same number of bits to an interval of one or more pictures despite their different encoding difficulties and produces visual quality that varies with the video content. For both VBR and CBR encoding systems, an encoder can allocate bits according to perceptual models within a picture. One characteristic of human perception is texture masking, which explains why human eyes are more sensitive to loss of quality at the smooth regions than in textured ones. This property can be utilized to increase the number of bits allocated to the smooth regions to obtain high visual quality.
Quantization process in a video encoder controls the number of encoded bits and the quality most intimately. It is common to adjust the quality through adjusting the quantization parameters. In the following, we use H.264/AVC as the example to explain the quantization process. Other standards, such as H.263 and MPEG-2 follow similar procedures. Mathematically, in the encoder the transformed coefficient W is quantized as:
                              Z          =                                    ⌊                                                                                        W                                                        q                                +                s                            ⌋                        ·                          sgn              ⁡                              (                W                )                                                    ,                            (        1        )            where Z is the quantization level. Here, q is the quantization step size and s is the quantization rounding offset. The function └.┘ rounds a value to the nearest integer and sgn(.) returns the sign of a signal. When the quantization matrix is applied, the coefficients are scaled first before the quantization process at the encoder. The range of W where it is quantized to 0 is called the deadzone. In this particular case, the deadzone is =Δ=(1−s)×qx 2 while the deadzone range is (−(1−s)×q, (1−s)×q). At the decoder, the quantization level Z is reconstructed to the signal W′. This is called inverse quantization and is described mathematically as:W′=q·Z.  (2)
The syntax in H.264/AVC allows q to be different for each macroblock (MB). The value of q is selected from the ones indexed by parameter QP, an integer in the range of 0-51. The rounding offset parameter s, is not involved in the inverse quantization and the encoder has the flexibility of setting it to any value.
Existing MPEG-4 AVC video encoders usually assume the quantization rounding offset is constant and only adjust the quantization step size to adjust the number of bits and therefore the quality. As can be seen from (1), the rounding offset has pronounced control over the small coefficients as it directly controls how the small near-zero coefficients are quantized. When we increase s, fewer coefficients are quantized to zeros and more bits are spent on the small coefficients given the quantization step size q. When the rate is given, an increased s needs to operate with a coarser q to meet the bit rate constraints. Therefore increasing the rounding offset may preserve the small coefficients at the cost of more distortions to the large coefficients. Since preserving small coefficients preserves fine details, including but not limited to film grain and computer-generated noise, in reconstructed video, adjusting rounding offset values can be very effective in obtaining high perceptual quality for some applications, such as Blu-Ray DVD authoring where transparent visual quality is expected.